fix: dj.Top is honored when a limited expression is used in another query - #1657

Fixes #1656. Fixes #1666 (a duplicate of #1656).

The bug

An expression restricted by dj.Top keeps its ORDER BY/LIMIT in _top, separately from its WHERE conditions. Several operations that take an expression as an operand ignored _top, or broke on it. They returned too many rows without a warning, or failed with a SQL error. All of the following reproduce on master on MySQL 8.0 and PostgreSQL. top_child selects one child, and top_parent selects parent 1.

The fix

The rule is the one #1650 applies to a restricted left-join operand: a limited operand is wrapped in a subquery, so its ORDER BY/LIMIT applies to that operand on its own.

• join(): an operand with _top, on either side, is a subquery. This covers *, aggr (inner and left), and extend.
• make_condition: a limited restricting expression is a subquery: IN (SELECT a FROM (SELECT … ORDER BY … LIMIT n) AS t). This is the usual MySQL workaround for 1235, and it applies the limit before the projection to the common attributes. That projection matters on PostgreSQL too, for example when the order is on a computed attribute.
• dj.U & expr and dj.U.aggr(group): a limited operand is a subquery, so the limit applies before DISTINCT/GROUP BY.
• Union.make_sql: sorting_clauses() moves out of the alias name and follows it. The branch for operands with secondary attributes is wrapped the same way when the union is limited.
• proj(): adds a subquery only when the ORDER BY names an attribute that the projection redefines (a rename or computation into that name) or a computed/renamed attribute that it drops. Dropping a plain table column needs no subquery, because ORDER BY on an unselected column is valid. So fetch(attr, order_by=..., limit=...) keeps its ordering at the outer level.
• Top.merge: the second Top skips other.offset rows of the first Top's window and takes up to other.limit, never past the first Top's limit. Previously min(limit1, limit2) with added offsets could reach past it.
• QueryExpression.__bool__: uses a subquery when _top is set, as __len__ already does.
• Diagram.restrict(): builds its seed with _seed_restriction(), as cascade and trace do. A dj.Top seed, or a seed restricted by another query, becomes the primary keys it selects, and % in seed conditions is no longer escaped twice when the conditions are re-applied.
• Heading.make_subquery_heading(): keeps table_info, so a rename after a subquery quotes for the backend. Without it, PostgreSQL got backticks. This is the same fix fix(heading): keep table_info in set_primary_key so a later rename quotes for the backend #1643 makes in set_primary_key, and it was needed once proj() started wrapping.

A - dj.Top() stays unsupported, as discussed in #1656. A negated or nested Top (-, dj.Not, a list, an AndList, or populate()/progress() restrictions) now raises:

dj.Top must be applied on its own, as expression & dj.Top(...). It cannot be negated (with - or dj.Not) or combined with other restrictions in a list, an AndList, or the restrictions passed to populate() or progress(). For the rows after the first n, use dj.Top(None, order_by=..., offset=n).

TestDjTop::test_top_errors previously pinned the old "Invalid restriction type Top(...)" text and is updated.

Behavior changes (for the v2.3.7 notes)

All of these bring results in line with what the queries say:

• Queries in the table above return the limited rows instead of the full set.
• dj.U(attrs) & (expr & dj.Top(n, ...)) limits expr first and then takes distinct values. The previous "first n distinct values" is written (dj.U(attrs) & expr) & dj.Top(n, ...).
• Chained Tops and to_dicts(limit=…, offset=…) on a limited expression no longer return rows past the first limit.
• Diagram.restrict() seeds the same way as cascade.

Tests

• tests/integration/test_top_in_queries.py: 49 cases × MySQL and PostgreSQL.
  • Joins: the limit on either side, on both sides, with a condition, the offset form, and a projected operand.
  • Left joins: extend and default aggr, with the limit on either side.
  • Aggregation: inner aggr and dj.U.aggr, with both Top and the offset form.
  • Restriction and anti-restriction by a limited expression: Top, Top(2), the offset form, the order on a computed attribute, and a limited union.
  • dj.U restriction: the order on a dropped attribute and on a kept one.
  • Union: restricted by Top, and of limited operands, with and without secondary attributes.
  • proj() of a limited expression: keys() of a limited aggregate, a dropped computed ordering attribute, an ordering attribute recomputed or swapped by a rename, and a dropped table column, which needs no subquery.
  • Chained limits, bool() and Diagram.restrict().
  • The negated/nested Top error: five forms.
  • Controls: no subquery for unlimited joins, restrictions, dj.U & and dj.U.aggr.
• tests/unit/test_condition.py: Top.merge windows.
• Each change is tested: reverting any one of the 13 source changes on its own fails at least one test. That covers both backends wherever the bug shows on both; the table_info change shows only on PostgreSQL.
• Full suite locally: 1323 passed, 14 skipped.

Interaction with other PRs

This branch merges cleanly with #1655, #1643, #1653 and #1654 (checked with git merge-tree). After #1655 merges, its or other._top in the left-join condition is redundant with the general rule here, but harmless.

Not changed here

These turned up in review. They are not about dj.Top and predate this PR:

• Restricting by a union, inserting from a union, or nesting a union with secondary attributes raises TypeError #1658: Union.make_sql() takes no fields argument, but make_condition and Table.insert pass one. So restricting by, or inserting from, an unlimited union raises TypeError. A limited union works after this PR, because it goes through a subquery.
• dj.U(...).aggr gives wrong results or fails when the group is a union, an aggregation, or a computed projection #1661: dj.U.aggr over a Union or an Aggregation group uses the group's support/restriction directly, dropping the second union operand or the inner GROUP BY.
• Joining with a union on the right raises AssertionError when the two sides share no attributes #1659: join() checks isinstance(self, Union) where it means other for the right operand.
• Joining with a dj.U restriction returns duplicate rows and inflates aggr counts: the DISTINCT is dropped #1660: Joining with a dj.U & … restriction can return duplicate rows, because the join ignores _distinct.